Query Understanding in Web Search - by Large Scale Log Data Mining and Statistical Learning

Query understanding is an important component of web search, like document understanding, query document matching, ranking, and user understanding. The goal of query understanding is to predict the user’s search intent from the given query. Needless to say, search log mining and statistical learning are fundamental technologies to address the task of query understanding. In this talk, I will first introduce a large-scale search log mining platform which we have developed at MSRA. I will then explain our approach to query understanding, as well as document understanding, query document matching, and user understanding. After that, I will describe in details about our methods for query understanding based on statistical learning. They include query refinement using CRF, named entity recognition in query using topic model, context aware query topic prediction using HMM. This is joint work with Gu Xu, Daxin Jiang and other collaborators.